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Abstract 


The  Federal  contracting  process  should  enable  a  government  organization  to  select  a  contractor 
that  will  become  a  true  business  partner.  Today’s  source  selection  processes  provide 
opportunities  to  evaluate  how  well  a  contractor  proposes  a  solution;  however,  the  government’s 
processes,  policies,  and  tools  are  ill  suited  to  evaluating  how  well  a  contractor  can  be  expected  to 
deliver  on  its  proposed  solutions.  Like  most  government  agencies,  the  Department  of  Defense 
(DoD)  relies  too  heavily  on  the  contractor’s  proposal — what  the  contractor  claims  it  can  do — 
versus  evaluating  past  performance  to  determine  what  a  contractor  has  proven  it  can  do.  The  lack 
of  adequate  past  performance  data  and  processes  to  effectively  evaluate  the  qualifications  of 
companies,  including  examples  of  the  contractor’s  trustworthiness  and  key  personnel,  has 
contributed  to  a  series  of  program  failures,  cost  overruns,  and  schedule  delays.  Without  adequate 
data  and  processes  to  address  these  issues,  the  DoD  increases  their  risk  of  duplicating  previous 
program  failures  and  missing  the  opportunity  to  capture  this  information  to  prevent  repeated 
mistakes  with  the  same  contractor. 

Two-line  summary 

Selecting  the  right  contractor  is  perhaps  the  most  critical  factor  in  achieving  future  program 
success.  This  research  identifies  past  performance  strategies  to  maximize  favorable  program 
outcomes. 


ii 


1. 


Introduction 


Selecting  the  right  contractor  is  perhaps  the  most  critical  factor  in  achieving  future  program 
success.  This  research  identifies  processes  and  proposes  strategies  that  could  strengthen  how  the 
government,  specifically  the  Department  of  Defense  (DoD),  selects  an  industry  partner  to 
maximize  the  probability  of  program  success. 

Ideally,  the  relationship  between  the  government  and  a  contractor  should  be  a  genuine  business 
partnership.  When  awarding  a  contract  for  a  major  weapon  system  or  program,  especially  one 
that  requires  significant  development,  the  DoD  must  expect  the  relationship  to  be  a  long-term 
commitment.  The  contractor  and  government  program  management  office  (PMO)  each  play 
numerous  critical  roles  throughout  the  program  planning,  executing,  and  reporting  cycles.  In  a 
successful  program,  a  contractor  organization  and  the  PMO  work  in  harmony  to  strive  toward 
program  success.  They  understand  and  leverage  each  other’s  strengths;  they  communicate 
frequently;  they  collectively  manage  the  vast  web  of  program  stakeholders;  and  they  collectively 
establish  the  reputation  and  credibility  of  the  program. 

The  government  must  therefore  strive  to  enter  into  business  relationships  that  provide  high 
confidence  of  success.  Selecting  an  experienced  and  qualified  business  partner  is  a  critical  step  in 
achieving  this  objective.  While  evaluation  of  the  contractor’s  proposal  is  important,  this  review 
can  provide  only  a  prediction  of  what  may  happen  and  a  description  of  how  the  contractor  plans 
to  perform.  By  contrast,  the  contractor’s  experience  and  past  performance  offers  an  objective 
measure  of  what  the  firm  has  actually  accomplished  in  the  past. 

The  paper  identifies  three  key  problem  areas: 
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1.  DoD  evaluates  the  contractor’s  proposed  solution  rather  than  the  contractor’s 
record  of  actual  performance.  During  the  source  selection  process  DoD  evaluates  a 
contractor’s  proposal,  which  consists  of  cost,  technical,  and  past  performance  volumes.  In 
most  cases,  however,  past  performance  has  little  or  no  impact  on  the  final  selection  of  a 
contractor.  Instead,  this  aspect  is  often  minimized  as  a  superficial  pass/fail  test  rather 
than  being  viewed  as  an  effective  tool  to  predict  how  well  a  contractor  will  perform  in  the 
future.  Without  objective  information  on  past  performance  and  processes  to  evaluate 
contractor  qualifications,  DoD  is  reluctant  to  discriminate  among  proposals  based  on  past 
performance,  especially  given  the  risk  of  protest. 

2.  DoD  lacks  consistent  and  thorough  processes  to  evaluate  key  personnel.  The 

personnel  on  the  contractor  program  team  make  vital  contributions  to  program  success, 
and  as  such,  their  qualifications  should  be  an  important  evaluation  factor.  Contractors 
generally  identify  key  personnel,  and  include  their  resumes  as  part  of  the  proposal 
package.  However,  most  source  selections  limit  evaluation  of  these  key  personnel  to  a 
simple  determination  of  whether  the  proposed  team  members  meet  the  minimum 
experience  criteria. 

3.  DoD  lacks  adequate  tools  to  collect,  analyze,  and  report  past  performance 
information  consistently  across  contracts.  The  ability  to  evaluate  past  performance 
effectively  depends  heavily  on  the  availability  of  tools  and  the  quality  of  data  content. 

The  government  is  currently  improving  and  consolidating  its  repositories  of  past 
performance  data;  however,  the  DoD  lacks  capabilities  for  information  collection  and  has 
not  consistently  enforced  data  entry  requirements. 
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These  problem  areas  represent  three  different  deficiencies  in  the  evaluation  of  a  contractor’s  past 
performance.  Evaluation  is  hampered  by  the  lack  of  adequate  tools  to  collect,  analyze,  and  report 
relevant  information.  The  lack  of  processes  to  assess  the  capabilities  and  experience  of  key 
personnel  puts  at  risk  the  government’s  ability  to  predict  future  contract  perfonnance.  In 
addition,  the  ability  to  collect,  analyze,  and  report  information  on  past  performance  is  impaired 
by  the  lack  of  policies  and  processes  to  incentivize  better  reporting.  Together,  these 
shortcomings  affect  the  overall  contracting  process,  and  contribute  to  a  cycle  of  inadequacies  that 
has  resulted  in  the  numerous  contracting  challenges  we  face  today.  The  DoD  needs  to  take  a 
comprehensive,  coherent  approach  to  resolve  these  problems  in  order  to  achieve  overall  past 
performance  improvements. 

1.1  Background 

With  the  passage  of  the  Federal  Acquisition  Streamlining  Act  of  1994,  Congress  acknowledged 
that  it  is  appropriate  and  relevant  for  the  government  to  consider  a  contractor’s  past  performance 
in  evaluating  whether  that  contractor  should  receive  future  work.  (U.S.  Department  of  Defense, 
2008) 1  In  response,  federal  departments  initiated  procedures  and  systems  to  record  and  use 
information  on  past  contractor  performance  during  source  selection.  Although  the  government 
recognizes  that  systematically  documenting  the  contractor’s  performance  becomes  a  powerful 
motivator  for  a  contractor  to  sustain  high-quality  outcomes,  a  consistent  process  for  recording 
performance  has  been  difficult  to  instantiate  and  maintain.2 


1  U.S.  Department  of  Defense.  (February  29,  2008).  Contractor  Past  Performance  Information  (Inspector  General  Report  No.  D- 
2008-057).  Washington,  DC:  U.S.  Department  of  Defense,  p.  i. 

2  Office  of  Federal  Procurement  Policy,  Office  of  Management  and  Budget,  Executive  Office  of  the  President.  (May  2000).  Best 
Practices  for  Collecting  and  Using  Current  and  Past  Performance  Information.  Washington,  DC:  Office  of  Management  and 
Budget.,  p.  1. 
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In  May  2010  the  Office  of  Federal  Procurement  Policy  (OFPP)  directed  government  agencies  to 
integrate  past  performance  data  systems  to  provide  consistency  and  ensure  reliability  of  data 
across  the  federal  government.  The  Acquisition  Committee  for  eGovernment  (ACE)  determined 
that  the  entire  federal  government  will  use  the  DoD  Contractor  Performance  Assessment 
Reporting  System  (CPARS)  as  the  single  system  for  collecting  and  transmitting  performance 
evaluations  to  the  Past  Performance  Information  Retrieval  System  (PPIRS). 3  The  Navy  created 
CPARS  in  1998  to  meet  information  requirements  established  by  the  Federal  Acquisition 
Regulation  (FAR).  Due  to  inconsistent  CPARS  data  collection  and  reporting,  DoD  published  a 
best  practices  guide  titled  “A  Guide  to  Collection  and  Use  of  Past  Performance  Information”  in 
May  2003.  In  June  2007,  DoD  issued  another  CPARS  Policy  Guide  that  required  all  new 
contracts  within  a  certain  dollar  threshold  to  register  in  CPARS  within  30  days  of  contract 
award.4 

On  January  21,  201 1,  the  former  OFPP  Administrator,  Daniel  Gordon,  sent  a  memorandum  to 
Chief  Acquisition  Officers  and  Senior  Procurement  Executives  on  enhancing  assessments  of 
contractor  past  performance,  which  included  steps  and  strategies  for  improving  the  collection  of 
information.  The  memorandum  emphasized  that  while  compliance  with  reporting  requirements  is 
important,  the  quality  of  the  reports  submitted  is  far  more  crucial.  It  also  acknowledged  various 
challenges  that  contribute  to  the  low  number  and  quality  of  these  assessments,  which  include 
staff  shortages  and  the  transition  to  the  new  federal- wide  system  that  integrates  the  PPIRS, 
CPARS,  and  the  Federal  Awardee  Performance  and  Integrity  Information  System  (FAPPIS).5 


3  Past  Performance  Information  Retrieval  System.  Retrieved  March  7,  2012,  from  http://www.ppirs.gov 
http://www.ppirs.gov/ 

4  Contractor  Past  Performance  Information,  p.  5. 

5  Gordon,  D.  I.  (January  21,  201 1).  Improving  Contractor  Past  Performance  Assessments:  Summary  of  the  Office  of  Federal 
Procurement’s  Policy’s  Review,  and  Strategies  for  Improvement.  Washington,  DC:  Office  of  Federal  Procurement  Policy,  Office 
of  Management  and  Budget,  Executive  Office  of  the  President. 
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Despite  efforts  by  DoD  and  other  government  organizations,  the  policies  and  mandates  lack 
enforcement  mechanisms  to  incentivize  better  behavior  and  improve  past  performance  practices. 
This  paper  recommends  solutions  that  will  strengthen  the  data,  processes,  and  tools  associated 
with  evaluating  past  performance  and  provide  the  government  with  a  strategy  to  benefit  from  the 
implementation  of  these  resources. 

2.  Problem:  DoD  Evaluates  the  Contractor’s  Proposed  Solution  vs.  Evaluating  the 
Contractor’s  Record  of  Actual  Performance 

FAR  Part  15.3  covers  the  selection  of  a  contractor  in  competitive  negotiations.  According  to  the 
FAR,  the  government  is  required  to  evaluate  three  areas  for  acquisitions:  the  quality  of  the 
proposed  product  or  service,  the  price  or  cost  to  the  government,  and  past  performance  (for 
acquisitions  that  exceed  the  simplified  acquisition  threshold).  The  government  often  requests  the 
contractor  to  submit  volumes  covering  technical,  cost,  and  past  performance  as  part  of  its 
proposal.  The  technical  proposal  conveys  the  contractor’s  proposed  technical  solution  or 
response  to  a  requirement;  the  cost  proposal  identifies  the  proposed  costs  of  delivering  the 
proposed  technical  solution;  and  the  past  performance  volume  shows  how  the  contractor 
performed  on  previous  similar  or  related  efforts  as  an  early  indicator  of  potential  future 
performance. 

Unfortunately,  the  DoD  currently  lacks  both  adequate  data  on  past  performance  and  effective 
processes  to  evaluate  a  company’s  qualifications,  including  key  personnel.  Thus,  evaluation  of 
past  performance  is  not  used  as  effectively  as  it  could  be  to  predict  future  performance  on  a 
contract.  These  conclusions  are  borne  out  by  a  2009  study  by  the  U.S.  Government 
Accountability  Office  (GAO).  The  study  examined  past  performance  practices  by  analyzing  62 
procurements  in  five  agencies  that  perform  major  acquisitions:  the  DoD,  the  Department  of 
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Energy  (DOE),  the  General  Services  Administration  (GSA),  the  Department  of  Homeland 
Security  (DHS),  and  the  National  Aeronautics  and  Space  Administration  (NASA).  The  study 
found  that  82  percent  of  the  past  performance  evaluations  did  not  contain  narratives  sufficiently 
detailed  to  establish  that  the  resulting  ratings  were  credible  or  justifiable. 6 

In  many  of  these  acquisitions,  the  government  cited  a  company’s  technical  approach  as  the  most 
important  non-cost  factor.  More  than  60  percent  of  the  contracting  officers  “stated  that  past 
performance  is  rarely  or  never  a  deciding  factor  in  selecting  a  contractor.”7  However,  the 
acquisitions  that  placed  emphasis  on  past  performance  noted  that  this  encouraged  companies  to 
perform  better.8  The  report  succinctly  concluded:  “Regardless  of  the  source  used,  contracting 
officials  agreed  that  for  past  performance  information  to  be  meaningful  in  contract  award 
decisions,  it  must  be  documented,  relevant,  and  reliable.”9 

Without  such  information,  the  government  evaluation  must  rely  heavily  on  factors  (primarily 
technical  and  cost)  in  the  contractor’s  proposal  that  describe  a  hypothetical  situation:  how  the 
contractor  plans  to  perform  on  a  contract.  In  essence,  this  means  that  evaluations  are  based  on  the 
quality  of  a  document  produced  by  a  professional  proposal  writing  team  during  the  45-90  day 
solicitation  timeframe.  These  teams  specialize  in  showcasing  their  company’s  capabilities;  they 
are  not  necessarily  experts  in  devising  solutions  within  the  company’s  technical  capabilities,  and 
that  the  company  can  execute  with  acceptable  risk.  Proposals  are  often  written  with  little  or  no 
input  from  the  staff  members  who  will  execute  the  day-to-day  contract  requirements.  As  a  result, 
proposals  submitted  for  competitive  source  selections  often  present  “optimistic”  solutions  that 
carry  significant  inherent  risk.  This  risk  often  materializes  during  contract  execution,  which 

6  U.S.  Government  Accountability  Office.  (April  2009).  Better  Performance  Information  Needed  to  Support  Agency  Contract 
Award  Decisions  (GAO-09-374).  Washington,  DC:  Government  Accountability  Office,  pp.8-9. 

7  Ibid,  pp.  8-9. 
s  Ibid,  pp.  2-3. 

9  Ibid,  pp.  8-9. 
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directly  contributes  to  program  failures,  cost  overruns,  and  schedule  delays,  which  are  endemic 
to  DoD  acquisitions.  The  heavy  scrutiny  of  technical  and  cost  proposal  evaluations  can  prove  to 
be  a  wasted  exercise  when  contractors  continually  overrun  budgets  or  require  major  engineering 
change  proposals  (ECPs)  throughout  the  life  of  the  acquisition. 

Given  the  shortcomings  described  above,  the  evaluation  of  past  performance  often  becomes  a 
superficial  pass/fail  test.  The  majority  of  best  value  source  selections  require  the  government  to 
evaluate  past  performance  on  the  basis  of  a  sliding  scale  that  ranges  from  high-confidence  to  no- 
confidence.  The  DoD  uses  this  scale  to  assess  its  level  of  confidence  in  a  contractor’s  ability  to 
perform  based  on  previous  related  experience.  Without  adequate  information  about  all  offerors, 
the  DoD  is  reluctant  to  place  strong  emphasis  on  past  performance,  especially  due  to  the  high 
risk  of  protest.  As  a  result,  the  DoD  usually  assigns  all  offerors  a  similar  confidence  rating  in 
order  to  “level  the  playing  field,”  and  uses  other  evaluation  factors  to  discriminate  among 
proposals.  Therefore,  not  surprisingly,  the  government  often  fails  to  select  a  business  partner 
with  an  established  history  of  high  performance  and  relevant  experience. 

The  Office  of  Federal  Procurement  Policy  (OFPP)  conducted  a  pilot  test  with  30  contracts  that 
used  past  performance  as  a  significant  evaluation  factor.  Those  contracts  identified  a  20  percent 
increase  in  average  customer  satisfaction,  confirming  the  assertion  that  the  increased  use  of  past 
performance  data  supports  positive  program  results.10  To  improve  how  the  DoD  leverages  past 
performance  in  its  source  selections,  the  remainder  of  this  section  reviews  proposed  solutions 
that  incorporate  well-founded  best  practices  to  mitigate  the  risks  associated  with  inadequate  past 
performance  evaluations. 


10  U.S.  General  Services  Administration,  Strategic  IT  Analysis  Division.  (December  1997).  Past  Performance  (White  Paper). 
Washington,  DC:  Office  of  Management  and  Budget.,  p.  2. 
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2.1  Proposed  Solutions 


2.1.1  Alpha  Contracting  Negotiations  with  Pre-Qualified  Vendor  Candidates 

Alpha  contracting  negotiations  with  pre-qualified  vendors  leverage  the  best  practices  of  the 
Qualifications-Based  Selection  (QBS)  strategy  authorized  for  Architecture  and  Engineering 
(A/E)  contracts  under  FAR  Part  36.  The  QBS  strategy  was  originally  developed  because  certain 
creative  professional  services  cannot  be  fairly  priced  before  the  creative  process  has  taken  place. 
The  QBS  process  improves  innovation,  quality,  and  creativity  by  allowing  the  government  to 
negotiate  the  contract  requirements  and  pricing  jointly  with  the  contractor  against  an  independent 
cost  estimate.  The  Brooks  Act  of  1972  delineates  the  framework  for  QBS  using  the  following 
phased  approach: 

1.  The  government  determines  the  desired  qualifications,  including  the  experience  and 
demonstrated  competence  of  interested  parties.  The  information  provided  by  the  offerors 
should  emphasize  technical  ability  (cost  is  not  considered  as  part  of  the  initial 
evaluation). 

2.  The  government  creates  a  short  list,  ranking  the  bidders  by  their  qualifications. 

3.  The  government  conducts  interviews  with  the  firms  on  the  short  list  and  then  re-ranks  the 
firms. 

4.  Finally,  the  government  negotiates  a  statement  of  work  (SOW)  and  fair  price  with  the 
most  qualified  firm. 

Unfortunately,  the  FAR  only  allows  use  of  QBS  for  contracts  involving  the  construction  or 
building  trades,  including  transportation  systems.  However,  the  DoD  could  leverage  some  of  the 
benefits  offered  by  the  A/E  procurement  strategy  by  combining  some  QBS  attributes  with  other 
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contracting  best  practices  and  strategies.  For  example,  DoD  can  use  a  prequalification  process  to 
narrow  the  pool  of  qualified  vendors,  similar  to  step  1  in  the  QBS  process.  This  can  be  done  by 
using  an  advisory  multi-step  process  described  in  FAR  Part  15.202,  or  by  establishing  a 
competitive  range  based  entirely  on  past  performance  criteria.  Narrowing  the  field  of  vendor 
candidates  would  ensure  that  the  DoD  optimizes  its  time  and  resources  by  conducting  Request 
for  Proposal  (RFP)  activities  only  with  the  vendors  that  have  the  highest  likelihood  of  being 
selected  for  contract  award.  This  approach  also  benefits  potential  contractors,  since  they  do  not 
have  to  expend  valuable  proposal  development  funds  for  solicitations  they  have  little  chance  of 
winning. 

Processes  to  narrow  down  the  pool  of  qualified  vendors  on  the  basis  of  past  performance  and 
experience  can  require  upfront  time  and  investment  from  the  government.  However,  these 
strategies  can  save  significant  time  during  the  “back-end”  of  the  proposal  evaluation  process. 
Ranking  the  vendors  also  helps  keep  competitive  pressures  in  play  during  negotiations.  The 
government  mitigates  the  disadvantages  associated  with  a  sole-source  negotiation  environment 
because  it  maintains  the  flexibility  to  move  to  the  next-ranked  vendor  at  any  point  in  the 
negotiations. 

Additionally,  DoD  can  use  a  question  and  answer  session  as  part  of  an  oral  presentation  strategy, 
similar  to  the  QBS  interview  process  described  in  step  3  above,  and  in  Section  3.1.1  of  this 
paper.  Open  communication  with  vendors  to  verify  experience  and  qualifications  is  an  important, 
but  often  overlooked  activity.  Usually  the  government  accepts  the  qualifications  and  experience 
stated  in  a  company’s  proposal  at  face  value.  However,  greater  insight  regarding  the  contractor’s 
related  experience  and  the  qualifications  of  the  proposed  key  personnel  would  increase  the 
accuracy  of  the  government’s  assessment. 
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Lastly,  DoD  can  apply  the  same  SOW  and  contract  price  negotiation  process  described  in  Step  4 
of  the  QBS  through  the  use  of  alpha  contracting:  an  innovative  strategy  that  allows  the 
government  to  perform  many  activities  jointly  with  the  contractor.  Together,  the  government  and 
contractor  develop  the  SOW  and  proposal  in  a  streamlined  fashion.  This  process  offers  a  number 
of  advantages  and  performance  improvements,  such  as  enhancing  communication,  refining  and 
clarifying  requirements,  and  ensuring  the  technical  solution  is  bounded  by  the  capabilities  of  the 
contractor.  The  alpha  contracting  process  also  has  the  potential  to  yield  significant  savings  in 
time  and  cost.  For  example,  the  Army  PM-Tank  Main  Armament  Systems  M830A1  procurement 
utilized  the  alpha  contracting  process  and  reduced  the  lead-time  for  procurement  administration 
by  55  percent,  thus  saving  $1  million,  which  was  subsequently  used  to  buy  additional  units.11 

It  may  be  possible  to  combine  all  of  the  above  techniques  and  use  a  General  Services 
Administration  (GSA)  Blanket  Purchase  Agreement  (BPA)  to  model  the  QBS  strategy  from  Step 

I  through  4.  Under  this  arrangement,  the  DoD  could  select  a  set  of  contractors  from  a  GSA 
schedule  and  use  a  prequalification  process  to  narrow  the  field  of  vendors.  Next,  the  DoD  would 
use  a  Q&A  process  to  interview  candidates  and  rank  order  the  potential  vendors.  The  DoD  could 
then  utilize  the  alpha  contracting  process  to  negotiate  a  BPA  with  the  highest  ranked  firm.  If 
negotiations  with  the  top-  ranked  vendor  resulted  in  unfair  pricing,  or  if  the  DoD  and  the  vendor 
were  unable  to  agree  on  a  SOW,  the  DoD  would  have  the  right  to  conduct  negotiations  with  the 
second-ranked  firm.  When  negotiations  are  complete,  the  DoD  would  issue  a  BPA  using  a  GSA 
schedule  contract  authorized  under  FAR  Part  8.405-3. 

This  GSA  BPA  approach  has  certain  restrictions  that  may  limit  applicability  across  a  broad  range 
of  acquisitions  (e.g.,  GSA  BPAs  are  limited  to  commercial  products  and  services  and  do  not 

II  Jones,  A.  (n.d.)  PROCNET.  PM-TMAS’s  M830A1  Alpha  Contracting  Process. 
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allow  cost-type  contracts).12  Furthermore,  the  QBS  approach  is  currently  limited  to  A/E  contracts 
authorized  under  FAR  Part  36.  However,  the  need  for  creativity  and  innovation  is  not  limited  to 
A/E  procurements.  Agile  software  development,  for  example,  values  flexibility,  innovation,  and 
collaboration  and  can  greatly  benefit  from  a  QBS-like  acquisition  approach. 

As  a  result,  the  DoD  should  pursue  an  amendment  to  FAR  Part  15  to  allow  a  QBS-like  process 
that  can  be  leveraged  for  many  different  types  of  acquisitions.  The  DoD  can  achieve  maximum 
benefit  from  this  approach  if  the  amendment  allowed  a  prequalification  process  to  narrow  the 
field  of  potential  vendors.  An  interview  or  Q&A  process  can  facilitate  the  vendor  ranking 
process,  and  alpha  contracting  would  allow  the  DoD  to  negotiate  a  SOW  and  pricing  based  on 
the  ranking  of  firms.  All  of  these  individual  strategies  are  permitted  under  different  areas  of  the 
FAR;  however,  a  FAR  Part  15  amendment  would  allow  the  DoD  to  combine  them  into  a  single 
process  to  obtain  maximum  benefit. 

2.1.2  Solution  Demonstrations  as  a  Source  Selection  Technique 

The  limited  availability  of  past  performance  information  makes  it  difficult  to  verify  the  relevance 
of  previous  experience.  As  an  alternative  strategy,  DoD  can  use  demonstrations  as  part  of  the 
source  selection  process  to  validate  previous  experience.  A  demonstration  could  include  any 
material  representation  of  past  experience  (e.g.,  prototypes,  software  demonstrations).  Using  this 
strategy,  the  DoD  would  require  vendors  to  submit  a  functioning  prototype,  or  a  previously 
developed  relevant  product,  and  to  demonstrate  its  capabilities  as  part  of  the  proposal  process. 
The  DoD  could  apply  this  strategy  to  both  hardware  and  software  procurements;  for  example, 
screen  mock-ups  and  software  demonstrations  from  previous  efforts  can  be  used  to  evaluate 

12  U.S.  General  Services  Administration,  U.S.  Department  of  Defense,  and  National  Aeronautics  and  Space  Administration. 
(January  2010).  Federal  Acquisition  Regulation  (FAR).  Washington,  DC:  Acquisition  Central.  Blanket  Purchase  Agreements, 
Subpart  13.303. 
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certain  software  procurements.  The  demonstration  could  be  submitted  prior  to  the  formal 
proposal  process,  or  could  be  used  to  supplement  a  written  technical  proposal.  The  proposal 
instructions  would  specify  the  desired  functionality  of  the  demonstrated  product,  and  thoroughly 
describe  how  the  DoD  would  evaluate  the  demonstration. 

This  source  selection  strategy  will  not  only  help  to  narrow  the  pool  of  qualified  vendors,  but  will 
also  offer  a  way  for  vendors  to  establish  recent  and  relevant  experience.  The  demonstration 
becomes  a  tangible  piece  of  evidence  that  the  contractor  has  verified  and  proven  its  experience 
with  the  technology  and/or  solution  proposed. 

3.  Problem:  DoD  Lacks  Consistent  and  Thorough  Processes  to  Evaluate  Key  Personnel 

The  overall  experience  and  business  processes  of  the  contractor  are  central  to  evaluation  of  past 
performance.  However,  the  qualifications  of  the  personnel  proposed  for  key  roles  on  the  contract 
have  even  greater  importance  for  the  future  success  of  the  program.  The  capabilities  and 
attributes  of  the  key  personnel  proposed  should  be  directly  relevant  to  the  successful  completion 
of  the  program’s  requirements  and  supported  by  the  strength  of  the  company’s  experience. 

DoD  processes  should  aspire  not  only  to  select  the  strongest  contractor,  but  also  to  ensure  that 
the  contractor  assigns  the  “A  Team”  to  the  program.  When  companies  know  their  key  personnel 
will  be  closely  evaluated,  they  have  an  incentive  to  offer  their  best  performers,  rather  than  simply 
personnel  who  meet  minimal  qualification  requirements.  Top  performing  personnel  with  a 
proven  track  record  will  also  be  inclined  to  ensure  the  technical  solution  proposed  can  be 
executed  with  acceptable  risk,  potentially  providing  internal  checks  and  balances  with  the 
proposal  writing  team. 
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Today,  contractors  generally  identify  key  personnel,  and  include  their  resumes  as  part  of  the 
proposal  package,  but  most  source  selections  limit  evaluation  of  these  key  personnel  to  a  simple 
check  of  whether  the  proposed  team  members  meet  experience  criteria.  The  DoD  should  have 
consistent  and  thorough  processes  for  the  evaluation  of  key  personnel  that  go  beyond  a  checklist 
of  experience  criteria.  This  paper  recommends  that  the  DoD  increase  the  use  of  oral 
presentations  and  Q&A  sessions  to  verify  the  qualifications  of  key  personnel. 

3.1  Proposed  Solution 

3.1.1  Evaluate  Key  Personnel  Using  Oral  Presentations 

Throughout  the  government,  oral  presentations  are  often  used  in  conjunction  with  written 
proposals  to  clarify  or  support  all  or  some  aspects  of  the  contractor’s  technical  and/or 
management  proposal.13  The  U.  S.  Air  Force  Source  Selection  Procedures  Guide  states  that 
“Discussions”  may  be  either  oral  or  written,  and  specifically  requires  such  discussions  for  those 
areas  of  the  proposal,  such  as  past  performance,  that  are  significant  enough  to  affect  the  source 
selection  decision.  These  presentations  can  greatly  enhance  the  evaluation  process.  They  give 
the  government  an  opportunity  to  evaluate  the  effectiveness  of  the  contractor’s  communication 
style,  clarify  the  contractor’s  proposal,  and  assess  important  attributes  and  group  dynamics  of  the 
contractor’s  key  personnel  team.  In  addition,  oral  presentations  remove  the  gloss  created  by 
professional  proposal  writers  and  give  the  government  an  opportunity  to  interact  directly  with  the 
contractor  team  that  will  perform  the  day-to-day  tasks  required  in  the  contract. 

The  DoD  should  take  advantage  of  this  opportunity  to  interact  directly  with  key  vendor 
personnel.  To  further  enhance  the  benefits  of  these  interactions,  the  DoD  should  mandate  that 


13  Sade,  M.  “Seven  Steps  to  Performance  Based  Acquisitions,”  Seven  Steps  Project,  www. acquisition,  gov  . 
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proposed  key  personnel  present  the  oral  proposals.  This  prevents  contractors  from  using  their 
best  marketing  representatives  to  make  presentations,  and  gives  the  government  an  opportunity 
to  evaluate  the  overall  strength  of  the  contractor  team  that  will  perform  the  actual  program  tasks. 

In  addition,  the  DoD  should  use  the  Q&A  portion  of  the  oral  presentation  to  verify  the 
qualifications  and  experience  of  key  personnel.  The  Air  Force  recommends  maximum  use  of  oral 
presentations  to  complement  or  replace  written  technical  volumes  where  they  best  fit  the 
acquisition. 14  The  DoD  can  review  the  resumes  of  the  proposed  personnel  ahead  of  time,  and 
prepare  targeted  questions  that  will  help  the  government  understand  the  breadth  and  depth  of  the 
team  member’s  actual  experience  relative  to  the  program’s  requirements.  The  government  should 
structure  questions  according  to  behavioral  interviewing  techniques:  asking  questions  that 
require  the  candidates  to  describe  a  past  situation  in  which  they  exhibited  a  specific  capability, 
such  as  problem  solving,  teamwork,  or  planning  and  organizing. 

The  premise  behind  behavioral  interviewing  is  that  the  most  accurate  predictor  of  future 
performance  is  past  performance  in  similar  situations.  This  strategy  is  used  by  more  than  70 
percent  of  Fortune  500  companies,  which  credit  it  with  being  55  percent  predictive  of  future  on- 
the-job-behavior,  while  traditional  interviewing  has  been  determined  to  be  only  10%  predictive.15 
This  method  of  interviewing  provides  two  types  of  information  that  improves  insight  regarding 
proposed  key  personnel:  1)  examples  of  how  they  actually  exhibited  the  desired  behavior,  versus 
a  hypothetical  discussion,  and  2)  evidence  of  the  success  or  failure  of  the  solution. 

DoD’s  evaluation  of  key  personnel  should  go  beyond  verifying  a  checklist  of  experience  criteria. 
Key  considerations  must  include  formal  assessments  of  the  contractor’s  trustworthiness  with 


14  U.S.  Air  Force.  (March  2000).  Source  Selection  Procedures  Guide.  Washington,  DC:  Office  of  the  Secretary  of  the  Air  Force.. 

15  Hansen,  K.  Quintcareers.com,  “Behavioral  Interviewing  Strategies  for  Job  Seekers.” 
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respect  to  foreign  influence  and  control,  the  rigor  of  corporate  security  protocols,  and  the 
integrity  of  the  contractor’s  global  supply  chain  relationships. 

Lastly,  the  contract  should  include  key  personnel  clauses  that  protect  the  government  from  “bait 
and  switch”  tactics.  In  some  instances,  contractors  have  replaced  the  team  originally  proposed 
with  potentially  less  qualified  candidates  shortly  after  contract  award.  To  mitigate  this  risk,  DoD 
contracts  should  require  that  the  key  personnel  originally  proposed  on  the  contract  remain  with 
the  program  for  a  minimum  time  period  (e.g.,  12-18  months),  unless  that  individual  leaves  the 
company.  This  becomes  especially  important  if  the  government  uses  key  personnel  as  an 
important  evaluation  factor  in  the  source  selection  process. 

4.  Problem:  DoD  Lacks  the  Adequate  Tools  and  Information  to  Collect,  Analyze,  and 
Report  Past  Performance  Information  Consistently  Across  Contracts 

A  high-quality  data  repository  greatly  strengthens  the  government’s  ability  to  evaluate  past 
performance  as  part  of  the  source  selection  process.  Well-documented  information,  easily 
accessible  in  such  repositories,  enables  the  government  to  defend  its  position  during  a  contract 
protest.  Thus,  a  data-driven  process  to  collect,  analyze,  and  report  past  performance  information 
is  critical  in  allowing  the  government  to  incorporate  past  performance  in  a  source  selection 
decision. 

As  noted  in  Section  1,  two  important  past  performance  tools  used  today  are  CPARS  and  PPIRS. 
CPARS  is  a  suite  of  web-enabled  applications  that  document  contractors’  past  performance  in 
accordance  with  the  FAR.  It  uses  electronic  workflow  to  automate  contracting  officers’ 
evaluations  of  contractor  performance.  It  also  allows  contractors  to  submit  comments 
electronically  in  response  to  a  government  assessment  and  either  concur  or  non-concur.  The 
DoD-developed  PPIRS  is  a  web-enabled  government  enterprise  application  supporting  source 
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selections.  Since  the  FAR  requires  all  federal  agencies  to  post  contractor  performance 
evaluations  in  PPIRS,  it  now  acts  as  the  government- wide  warehouse  for  information  on 
contractor  past  performance. 16  As  a  warehouse,  it  allows  query-based,  read-only  retrieval  and 
review  of  contractor  data  by  qualified  government  acquisition  personnel.  Contractors  can  view 
their  own  data  by  using  a  central  contractor  registration  to  gain  access. 

Although  the  FAR  requires  the  government  to  evaluate  past  performance,  the  collection  and 
reporting  of  past  performance  data  has  been  inconsistent  and  often  untimely.  The  lack  of 
consistent,  available,  and  reliable  data  hinders  the  DoD’s  ability  to  make  past  performance  an 
effective  part  of  the  proposal  evaluation  process.  The  government  has  recently  made 
improvements  to  many  of  the  past  performance  tools,  including  CPARS  and  PPIRS,  to  provide 
commonality  of  data  entry  format  and  increase  the  availability  of  data  for  retrieval.  Despite  these 
improvements,  current  data  repositories  still  perform  their  functions  inadequately,  or  are  used 
inaccurately  across  DoD.  The  problems  with  the  government’s  current  data  repositories  have 
plagued  DoD  for  years.  Poor  data  quality,  incompatible  data  formats,  inconsistent  reporting 
timeliness,  lack  of  data  entry,  diffused  accountability,  and  subjective  evaluations  are  among  the 
issues  associated  with  the  way  DoD  has  collected,  analyzed,  and  reported  past  performance 
information. 

4.1  Proposed  Solutions 

4.1.1  Implement  and  Enforce  DoD  Policies  for  Past  Performance 

The  OFPP  and  the  Defense  Procurement  and  Acquisition  Policy  (DPAP)  Directorate  in  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (AT&L) 

16  Information  found  at  http://www.ppirs.gov  However,  it  should  be  noted  that  while  all  federal  acquisitions  are  required  to  use 
the  PPIRS,  not  all  government  organizations  have  complied. 
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have  recently  issued  guidance  and  policy  that  emphasize  the  importance  of  information  on  past 
performance.  Current  policies,  however,  lack  appropriate  enforcement  mechanisms  to  make  this 
policy  effective  and  actionable.  AT&L/DPAP  should  therefore  issue  a  DoD-wide  mandate  to 
populate  past  performance  databases  in  a  timely  and  consistent  manner.  Per  FAR  Part  42.15,  the 
responsibility  for  reporting  past  performance  resides  with  the  contracting  officer,  but  it  is 
important  to  make  this  a  shared  responsibility  between  the  PMO  and  the  contracting  office.  The 
program  team  (e.g.,  program  manager,  contracting  officer’s  technical  representative  [COTR]) 
should  evaluate  how  well  the  contractor  is  performing  against  the  SOW  requirements.  The 
contracting  office  should  collect  this  information  from  the  PMO,  and  also  provide  input  on  the 
contractor’s  performance  against  the  contract  requirements  (e.g.,  reporting  requirements,  terms 
and  conditions).  The  contracting  office  should  have  ultimate  responsibility  for  collecting  this 
information  and  populating  the  past  performance  database  in  a  timely  and  consistent  manner. 

DoD  should  increase  enforcement  of  this  policy  through  a  variety  of  mechanisms.  Agencies 
should  be  required  to  report  metrics  on  how  well  they  are  complying  with  this  mandate. 
AT&L/DPAP  should  collect  these  metrics  on  a  monthly  basis  and  publish  the  results  on  DPAP’s 
website.  The  head  of  each  Combatant  Command,  Service,  or  Agency  (C/S/A)  should  be  held 
accountable  for  C/S/A-wide  compliance  with  this  policy. 

Additionally,  DoD  should  require  that  past  performance  reviews  become  a  mandatory  part  of 
acquisition  program  reviews  by  the  Program  Executive  Office  (PEO).  This  will  ensure  that 
program  offices  are  held  accountable  for  meeting  this  policy  requirement.  As  a  further  incentive, 
DoD  should  prohibit  programs  from  exercising  option  awards,  award  terms,  or  award  fee 
payouts  until  they  have  populated  past  performance  information  into  a  database. 
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Lastly,  DoD  should  include  an  evaluation  of  compliance  with  this  process  as  part  of  each 
contracting  officer’s  annual  pay  and  performance  review.  Metrics  that  report  compliance  with 
this  mandate  at  the  individual  contracting  officer  level  will  further  enhance  the  enforcement  of 
this  policy. 

4.1.2  Identify  Quantifiable  Metrics  for  Agency-Wide  Reporting 

Past  performance  evaluations  today  tend  to  be  based  on  subjective  judgments  by  the  evaluator. 
This  makes  it  difficult  to  compare  performance  across  programs  and  contractors.  Additionally, 
observations  show  that  program  managers  and  contracting  officers  are  often  reluctant  to  report 
negatively  on  past  performance  because  this  can  reflect  poorly  on  their  own  ability  to  manage  the 
program  or  contract.  Furthermore,  to  avoid  conflict  with  the  contractor,  the  government  may 
refrain  from  documenting  performance  deficiencies  in  official  databases.  As  a  result,  the  past 
performance  write-up  does  not  always  reflect  a  contractor’s  performance  accurately. 

The  DoD  should  agree  on  a  set  of  quantifiable  metrics  to  capture  objective  data  within 
standardized  past  performance  databases.  The  use  of  such  quantifiable  and  simple  yes/no  metrics 
can  provide  a  consistent  and  reliable  way  to  compare  contractor  performance  across  contracts. 
For  example,  earned-value  management  data  can  be  used  to  report  cost  and  schedule 
performance  on  complex  acquisitions.  Award  fee  determinations  can  also  provide  a  metric,  since 
the  award  fee  represents  a  bilateral  agreement  and  contractors  often  use  this  metric  to  measure 
their  own  performance.  Simple  yes/no  metrics  can  be  used  for  non-Earned  Value  Management 
(EVM)  contracts.  Such  questions  could  include:  Did  the  contractor  perform  within  proposed 
cost?  Did  the  contractor  deliver  on  time?  and  Did  the  program  encounter  a  Nunn-McCurdy 
breach?  Each  of  the  metrics  and  questions  can  be  followed  by  a  data  field  that  allows  the 
government  to  explain  the  evaluations  provided  in  greater  detail. 
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4.1.3  Include  Validated  and  Approved  Contract  Deliverables  in  Past  Performance 
Databases 

The  DoD  can  increase  the  quantity  and  quality  of  the  data  in  past  performance  databases  by 
incorporating  information  on  validated  and  approved  contract  deliverables.  For  example, 
summary  narratives  on  award  fees  provide  a  wealth  of  information  on  how  the  contractor 
performed  during  the  award  fee  period.  As  noted  above,  award  fees  reflect  consensus  between 
the  government  and  contractor  on  the  factors  that  contributed  to  the  final  fee.  This  type  of 
information  can  be  very  useful  during  evaluations  of  past  performance,  since  it  gives  the  DoD  a 
more  complete  picture  of  how  well  the  contractor  performed  over  the  life  of  a  contract. 

Lastly,  past  performance  databases  should  include  a  mechanism  for  uploading  documents 
directly  into  CPARS  to  share  with  other  government  counterparts.  This  would  greatly  enrich  the 
quality  and  quantity  of  past  performance  data. 

5.  Conclusion  and  Recommendations 

The  lack  of  adequate  past  performance  data,  tools,  and  processes  hinders  the  government  from 
effectively  evaluating  the  qualifications  of  companies  and  key  personnel.  The  DoD  tends  to 
deemphasize  past  performance  evaluations  during  the  competitive  source  selection  process, 
largely  because  data  that  would  allow  it  to  discriminate  among  proposals  is  unavailable.  Current 
government  tools  have  proven  inadequate  to  collect,  analyze,  and  report  information  on  past 
performance.  The  absence  of  timely,  reliable,  documented  data  impedes  the  entire  cycle  of 
collecting,  reporting,  and  evaluating  past  performance  during  source  selection  (see  Figure  1). 
The  inability  to  use  past  performance  as  a  predictor  of  future  contract  success  places  the  DoD  at 
risk  of  repeating  program  mistakes. 
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Past  Performance  Cycle 


Figure  1.  Past  Performance  Cycle 


Evaluating  a  company’s  performance  record  allows  the  government  to  assess  the  effectiveness  of 
a  contractor’s  management  processes,  tools,  and  resources.  Because  effective  performance 
depends  on  qualified  and  experienced  personnel  who  know  how  to  replicate  success,  the  ability 
to  evaluate  proposed  key  personnel  also  becomes  an  important  aspect  of  estimating  a  company’s 
ability  to  meet  the  contract  requirements.  Yet  the  DoD  is  not  utilizing  the  necessary  processes  to 
fully  assess  the  capabilities  of  the  key  personnel  proposed  on  a  contract. 

The  DoD  currently  lacks  a  comprehensive  approach  to  tackle  the  issues  identified  in  this  paper. 
Since  the  problems  are  interdependent,  they  require  a  holistic  approach  that  addresses  the  tools, 
processes,  and  policies  surrounding  the  past  performance  cycle  pictured  above.  The  processes 
will  not  improve  without  adequate  tools  to  provide  the  data.  They  will  not  be  effective  without 
the  proper  policies  in  place  to  enforce  them,  and  they  cannot  be  implemented  without  the  proper 
processes  and  tools  to  support  them. 

The  recommendations  summarized  below  have  been  brought  forward  throughout  this  paper  as 
important  steps  toward  improving  the  DoD’s  probability  of  achieving  successful  future  program 
outcomes: 
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1.  Use  alpha  contracting  negotiations  with  pre-qualified  vendor  candidates.  Amend  FAR 
Part  15  to  allow  for  a  QBS-like  approach  across  a  broader  range  of  acquisitions. 

2.  Use  solution  demonstrations  as  part  of  source  selection  to  increase  accuracy  in 
evaluations  of  the  contractor’s  performance  record. 

3.  Include  oral  presentations  in  the  evaluation  process;  improve  the  consistency  and  depth  of 
personnel  evaluations  by  using  the  behavioral  Q&A  process  to  verify  the  relevant 
qualifications  and  experience  cited  in  proposed  key  personnel  resumes. 

4.  Issue  a  policy-wide  mandate  that  enforces  past  performance  data  entry  standards.  Use  an 
online  tool  to  track  and  report  compliance  to  this  policy  requirement. 

5.  Agree  on  a  set  of  quantifiable  metrics  to  capture  objective  data  within  standardized  past 
performance  databases. 

6.  Increase  the  quantity  and  quality  of  data  in  past  performance  databases  by  incorporating 
relevant  contract  performance  documentation  from  validated  and  approved  contract 
deliverables. 
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BLUF 

Selecting  the  right  contractor  is  perhaps  the  most  important 

contributor  to  program  success. 

The  Past  Performance  as  an  Indicator  of  Future 
Performance  initiative  is  a  MITRE  Project  to  improve  the 
probability  of  a  successful  program  outcome. 

Emphasize  evaluating  the  contractor’s  record  of  performance  vs. 
evaluating  the  contractor’s  proposed  solution. 

-  Initiate  consistent  and  thorough  processes  to  effectively  evaluate 
proposed  key  contractor  personnel. 

-  Improve  the  accuracy  and  availability  of  government  of  past 
performance  data  repositories. 

The  recommendations  can  benefit  the  PMO  by  providing 
repeatable  data  driven  processes  using  historically  disparate 
best  practices  that  can  be  effectively  applied  to  past 
performance. 
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its  reserved 


The  government  must  strive  to  enter  into  business  relationships  that  provide 

high  confidence  of  success. 


MITRE 


Past  Performance  Project  Objectives 

Provide  approaches  to  improve  the  probability  of  selecting  an  industry 
partner  to  maximize  the  probability  of  program  success . 

Appraise  the  contractor’s  record  of  performance. 

Increase  Alpha  contracting  negotiations  with  pre-qualified  candidates. 

Consider  product  demonstrations  that  exhibit  performance  standards  early  in 
the  source  selection  process. 

Evaluate  proposed  key  contractor  personnel. 

Require  oral  presentations  using  an  extensive  question  and  answer  process  to 
verify  “actual”  qualifications  and  experience. 

Implement  behavioral  interviewing  techniques  to  identify  the  desired  attributes 
of  key  personnel’s  past  performance. 

Improve  timeliness  and  quality  of  past  performance  data  repositories. 

Enforce  past  performance  data  entry  standards. 

Improve  data  repository  tools  to  include  automated  quantifiable  metrics. 


Evaluating  contractor  past  performance  is  an  important  step  to  creating  a 
genuine  business  partnership  between  the  government  and  a  contractor. 
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Background 

Federal  Acquisition  Streamlining  Act  of  1994. 

Congress  acknowledged  that  it  is  appropriate  and  relevant  for  the  government 
to  consider  a  contractor’s  past  performance  in  evaluating  whether  that 
contractor  should  receive  future  work. 

The  Office  of  Federal  Procurement  Policy  (OFPP),  May  2010. 

Directed  government  agencies  to  integrate  past  performance  data  systems, 
Contractor  Performance  Assessment  Reporting  System  (CPARS)  and  the  Past 
Performance  Information  Retrieval  System  (PPIRS). 

Administrator  for  Federal  Procurement  Policy  Memorandum, 

Daniel  I.  Gordon,  January  21, 2011. 

-  “...ensuring  that  Contracting  Officers  have  access  to  meaningful  past 
performance  assessments  is  so  important  to  improving  source  selection 
decisions  that  we  want  to  do  everything  we  can  to  improve  both  the  quantity 
and  quality  of  past  performance  assessments  in  PPIRS.” 
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reserved 


The  policies  and  mandates  lack  enforcement  mechanisms  to  incentivize  better 

behavior  and  improve  past  performance  practices. 


MITRE 


Process  Overview 


Past  Performance  Cycle 
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Contractor  s  Record  of  Performance 

(Weakness) 

The  DoD  currently  lacks  both  adequate  data  and  effective  processes  to 
evaluate  a  company's  past  performance  qualifications . 

Government  Accountability  Office  Study,  April  2009. 

Analyzed  62  procurements  across  5  agencies  (DoD,  DOE,  GSA,  DHS,  and 
NASA). 

82%  of  past  performance  evaluations  did  not  contain  narratives  sufficiently 
detailed  to  establish  credible  or  justifiable  ratings. 

60%  of  the  Contracting  Officers  stated  that  past  performance  is  rarely  or  never 
the  deciding  factor  in  selecting  a  contractor. 

However,  the  study  concluded... 

Contracting  officials  agreed  that  for  past  performance  to  be  meaningful  in 
contract  award  decisions  it  must  be  documented,  relevant,  and  reliable. 
Acquisitions  that  placed  emphasis  on  past  performance  noted  that  this 
encouraged  companies  to  perform  better. 

Without  accurate  past  performance  information  the  government  must  rely 
heavily  on  technical  and  cost  factors  that  describe  a  hypothetical  situation. 


Raise  the  value  of  “past  performance”  in  the  source  selection  process. 
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Contractor  s  Record  of  Performance 

(Solution) 

Alpha  contracting  negotiations  with  pre-qualified  vendors. 

Utilizing  the  best  from  the  Qualifications  Based  Selection  (QBS)  (FAR  Part  36) 
to  improve  innovation,  quality,  and  creativity: 

The  government  determines  the  desired  qualifications,  experience  and 
demonstrated  competence  of  interested  vendors. 

Then  creates  a  short  list,  ranking  the  bidders  by  their  qualifications. 

The  government  conducts  interviews  with  the  firms  on  the  short  list  and  then 
re-ranks  the  firms. 

4.  Finally,  the  government  negotiates  a  statement  of  work  (SOW)  and  fair  price  with 
the  most  qualified  vendor  based  on  a  Government  Independent  Cost  Estimate. 

Much  of  QBS  can  be  applied  to  the  advisory  multi-step  process  (FAR  Part 
15.202)  based  entirely  on  past  performance  criteria  to  narrow  down  the  pool. 

Use  a  Q&A  session  as  part  of  an  oral  presentation  strategy. 

Performance  improvements  include  enhancing  communication,  refining  and 
clarifying  requirements,  and  bounding  the  technical  solution  by  the  capabilities 
of  the  contractor. 


Amend  FAR  Part  15  to  allow  DoD  to  directly  benefit  from  the  QBS  process. 
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Contractor  s  Record  of  Performance 

(Solution) 


Pre-qualification  product  demonstrations. 

Supplements  limited  availability  of  past  performance  information  and  verifies 
the  relevance  of  previous  experience. 

Demonstrations  can  include  any  material  representation  of  similar  conceptual 
past  experience,  such  as  prototypes  or  software  demonstrations. 

This  source  selection  strategy  assists  with  initial  narrowing  the  pool  of  qualified 
vendors. 

Establishes  recent  and  relevant  experience. 


Demonstrations  become  tangible  evidence  of  past  performance. 
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Key  Contractor  Personnel 

(Weakness) 

DoD  lacks  consistent  and  thorough  processes  to 
effectively  evaluate  key  contractor  personnel. 

Limited  evaluation  of  proposed  key  contractor  personnel. 

DoD  processes  do  not  emphasize  the  selection  of  the  strongest  contractor,  or 
ensures  that  the  contractor  assigns  the  “A  Team”  to  the  program. 

Most  source  selection  teams  limit  evaluation  of  key  personnel  to  a  simple 
checklist  to  whether  they  meet  minimum  “experience”  criteria. 

Current  minimal  use  of  oral  presentations  limit  face-to-face  access  to  key 
personnel  and  an  opportunity  to  assess  group  dynamics. 


When  companies  know  their  key  personnel  will  be  evaluated,  they  have 

an  incentive  to  offer  their  best  performers. 
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Key  Contractor  Personnel 

(Solution) 

Qualifications  of  personnel  proposed  for  key  roles  are  paramount 

to  the  future  success  of  the  program . 

Evaluate  proposed  key  contractor  personnel. 

U.S.  Air  Force  Source  Selection  Procedures  Guide  requires  discussions 
(written  or  oral)  for  proposal  areas  significant  enough  to  affect  the  source 
selection  decision.* 

Require  oral  presentations  using  an  extensive  question  and  answer  (Q&A) 
process  to  verify  “actual”  experience  and  qualifications  of  all  key  personnel. 

Use  behavioral  interviewing  techniques  to  evaluate  and  score  key  personnel’s 
past  qualifications  and  experience  against  program  requirements. 

-  Accept  only  the  BEST... 

‘Source:  USAF  Source  Selection  Procedures  Guide,  March  2000,  Section  2.9,  p.  33 


70%  of  Fortune  500  companies  credit  behavioral  interviewing  with  being  55% 
predictive  of  future  on-the-job  behavior  vs.  10%  for  traditional  interviewing  techniques. 
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Key  Contractor  Personnel 

(Solution) 


A  closer  look  at  behavioral  interviewing . 

Understanding  the  results  of  key  personnel’s  actual  past 
performance,  provides  greater  insight  of  future  success  during  the 
source  selection  process. 

Behavioral  interviewing  questions  are  targeted  on  prior  history  relevant  to  the 
requirements  of  the  future  program. 

This  method  of  interviewing  provides  two  types  of  important  information: 

Examples  of  how  they  actually  exhibited  the  desired  behavior,  i.e., 
technical,  problem  solving,  interpersonal,  leadership,  etc. 

Evidence  of  their  success  or  failure. 


Premise  of  Behavioral  Interviewing:  the  most  accurate  predictor  of 
future  performance  is  past  performance  in  a  similar  situation. 
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Data  Repositories 
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Data  Repositories 

(Weakness) 


DoD  Lacks  the  adequate  tools  and  information  to  collect,  analyze,  and 
report  past  performance  information  consistently  across  contracts. 

Despite  recent  improvements  to  systems  and  policies  current  data 
repositories  continue  to  perform  their  functions  inadequately. 

-  Lack  of  enforcement  of  past  performance  policies. 

-  Limited  quantifiable  metrics  for  agency-wide  reporting. 

-  Lack  of  validated  and  approved  contract  deliverables  in  the  past 
performance  databases. 


A  high-quality  data  repository  greatly  strengthens  the  government's  ability  to 
evaluate  past  performance  as  part  of  the  source  selection  process. 
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Summary  of  Agencies'  Past  Performance  Assessment  Review 
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Source:  Gordon,  Daniel,  Improving  Contractor  Past  Performance  Assessments  Memo  ,  OFPP,  January  21,  2011 


Average  High  59%  and  Low  13% 
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PAST  PERFORMANCE 

INFORMATION  RETRIEVAL  SYSTEM 


DoD  Reporting  Compliance  Metrics  2Q  FY2012 


AGENCY  NAME 

PERCENTAGE 

CONTRACTS 

COMPLETED 

3  JAN  2012 

PERCENTAGE 

CONTRACTS 

COMPLETED 

6  APR  2012 

QTR  TO  QTR 
DELTA 

DEPT  OF  THE  NAVY 

68.8 

46.8 

22- 

DEPT  OF  THE  ARMY 

53.7 

48.3 

5.4- 

DEPT  OF  THE  AIR  FORCE 

84.6 

57.3 

27.3- 

DEFENSE  HUMAN  RESOURCES  ACTIVITY 

87.5 

83.3 

4.2- 

TRICARE  MANAGEMENT  ACTIVITY  (TMA) 

72.7 

29.2 

43.5- 

DEFENSE  THREAT  REDUCTION  AGENCY  (DTRA) 

77.3 

52.5 

24.8- 

DEFENSE  CONTRACT  MANAGEMENT  AGENCY  (DCMA) 

69.1 

22.4 

46.7- 

USTRANSCOM 

83.9 

11.7 

72.2- 

BUSINESS  TRANSFORMATION  AGENCY 

31.5 

4.5 

27- 

DEFENSE  ADVANCED  RESEARCH  PROJECTS  AGENCY  (DARPA) 

73.6 

38.8 

34.8- 

DEFENSE  INFORMATION  SYSTEMS  AGENCY  (DISA) 

70.5 

31.6 

38.9- 

DEFENSE  LOGISTICS  AGENCY 

29.6 

24.6 

5- 

DEFENSE  SECURITY  COOPERATION  AGENCY 

7.2 

4.4 

2.8- 

DEFENSE  SECURITY  SERVICE 

42.8 

6.4 

36.4- 

DEFENSE  COMMISSARY  AGENCY  (DECA) 

5.6 

5.1 

.5- 

DEFENSE  FINANCE  AND  ACCOUNTING  SERVICES  (DFAS) 

84.1 

42.3 

41.8- 

DEFENSE  MEDIA  ACTIVITY  (DMA) 

13.1 

2.7 

10.4- 

DEPT  OF  DEFENSE  EDUCATION  ACTIVITY  (DODEA) 

4.1 

0 

4.1- 

WASHINGTON  HEADQUARTERS  SERVICES  (WHS) 

21.4 

5.1 

16.3- 

UNIFORMED  SERVICES  UNIVERSITY  OF  THE  HEALTH  SCIENCES  (USUHA) 

9.1 

0 

9.1- 

MISSILE  DEFENSE  AGENCY  (MDA) 

33.3 

13.7 

19.6- 

U.S.  SPECIAL  OPERATIONS  COMMAND  (USSOCOM) 

59.9 

16.5 

43.4- 

TOTAL  AVERAGES 

59% 

47% 

12%- 

Source:  Ginman,  Richard,  Contractor  Past  Performance  Assessment  Reporting-2nd  Quarter  2012  Memo,  April  26,  2012 
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Data  Repositories 

(Solution) 

Implement  and  enforce  DoD  policies  for  past  performance . 

Make  the  responsibility  for  reporting  past  performance  a  shared  responsibility 
between  the  contracting  office  and  the  program  management  office. 

The  contracting  office  should  have  ultimate  responsibility  for  collecting  and 
populating  the  past  performance  database  in  a  timely  and  consistent  manner. 

Agencies  should  be  required  to  report  compliance  metrics  against  this  mandate. 

DoD  should  further  require  that  past  performance  reviews  become  a  mandatory 
part  of  Program  Executive  Office  (PEO)  acquisition  program  reviews. 

DoD  should  prohibit  programs  from  exercising  option  awards,  award  terms,  or 
award  fee  payments  until  they  have  populated  past  performance  data  into  the 
database. 

Lastly,  DoD  should  include  an  evaluation  of  compliance  with  this  process  as  part  of 
each  contracting  officer’s  annual  pay  and  performance  review. 


Follow-up,  measure,  and  enforce  timely  and  accurate  past  performance  data  management. 
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Data  Repositories 


(Solution) 


Identify  quantifiable  metrics  for  agency-wide  reporting. 

Increase  the  accuracy  of  past  performance  evaluations. 

Develop  objective  measures  that  can  be  fairly  applied  to  all  contractors 
allowing  consistent  performance  comparisons  across  programs  and 
contractors. 

Provide  safeguards  for  program  managers  and  contracting  officers  from 
contractor  retaliation  through  multiple  source  evaluation  data  gathering  that 
encourages  honest  and  accurate  past  performance  evaluation  reporting. 

DoD  would  benefit  from  the  use  of  quantifiable,  and  simple  yes/no 
metrics  to  avoid  subjective  judgments,  and  introduce  consistency 
across  agencies. 
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Data  Repositories 

(Solution) 

Provide  access  to  validated  and  approved  contract  deliverables 

in  the  past  performance  database. 

The  DoD  can  increase  the  quantity  and  quality  of  the  data  in  past  performance 
databases  by  incorporating  information  on  validated  and  approved  contract 
deliverables. 

Provides  a  more  complete  picture  of  how  well  the  contractor  performed  over 
the  life  of  a  contract. 

Past  performance  databases  should  include  a  mechanism  for  uploading 
documents  directly  into  CPARS  to  share  with  other  government  agencies. 
This  would  greatly  enrich  the  quality  and  quantity  of  past  performance  data. 


Raise  the  value  of  “past  performance”  in  the  source  selection  process. 
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Summary 

The  lack  of  adequate  past  performance  data,  tools,  processes,  and  policy 
enforcement  hinders  the  government  from  effectively  evaluating  the  qualifications 

of  companies  and  key  personnel . 


Past  Performance  Cycle 
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Recommendations 

Implementation  of  the  following  recommendations  could  have 
a  significant,  positive  impact  on  future  program  success . 

Use  alpha  contracting  negotiations  with  pre-qualified  vendor  candidates.  Amend  FAR 
Part  15  to  allow  for  a  QBS-like  approach  across  a  broader  range  of  acquisitions. 

Use  product  demonstrations  as  part  of  source  selection  to  increase  accuracy  in 
evaluations  of  the  contractor’s  performance  record. 

Include  oral  presentations  in  the  evaluation  process;  improve  the  consistency  and  depth 
of  personnel  evaluations  by  using  the  Q&A  process  to  verify  the  qualifications  and 
experience  cited  in  proposed  key  personnel  resumes. 

Issue  a  policy-wide  mandate  that  enforces  past  performance  data  entry  standards.  Use 
an  online  tool  to  track  and  report  compliance  to  this  policy  requirement. 

Agree  on  a  set  of  quantifiable  metrics  to  capture  in  data  fields  within  databases  of  past 
performance. 

Increase  the  quantity  and  quality  of  data  in  past  performance  databases  by 
incorporating  information  on  validated  and  approved  contract  deliverables. 


Past  performance's  value  will  increase  with  efficient  tools  to  provide  the  data, 
effective  enforcement  of  policies,  and  supportinq  processes. 
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Past  Performance  is  an  indicator  of  Future  Performance 
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